A multi-expert approach for developing testing and diagnostic systems based on the concept-effect model

With the popularization of computer and communication technologies, researchers have attempted to develop computer-assisted testing and diagnostic systems to help students improve their learning performance on the Internet. In developing a diagnostic system for detecting students’ learning problems, it is difficult for individual teachers to address the exact relationships between the test items and the concepts. To cope with this problem, this study proposes an innovative approach to eliciting and integrating the weightings of test item-concept relationships from multiple experts. Based on the proposed approach, a testing and diagnostic system has been implemented; moreover, an experiment was conducted to evaluate the performance of our approach. By analyzing the results from four groups of students using learning suggestions provided by different models, it was found that the learning performance of the students who received learning suggestions by applying the innovative approach was significantly better than for those who received guidance based on the original model. 2010 Elsevier Ltd. All rights reserved. 1. Background and motivation With the popularization of computers and communication technologies, information aswell as instructional activities has been located on the Internet. Notable examples include the development of computer-assisted tutoring and testing systems (Chiou, Hwang, & Tseng, 2009; Hooper, 1992; Hwang, Cheng, Chu, Tseng, & Hwang, 2007; Hwang, Chu, Yin, & Lin, 2008; Hwang, Yang, Tsai, & Yang, 2009; Lee, Lee, & Leu, 2009; Plessis, Biljon, Tolmie, & Wollinger, 1995; Springer & Pear, 2008; Tsai & Chou, 2002; Tseng, Chu, Hwang, & Tsai, 2008). Specifically, during the tutoring process, online evaluation becomes an important formof learning feedback to provide the learning status of each student, and tests are mostly used as a way to gain this knowledge (Gronlund, 2003; Hwang et al., 2007; Tsai & Chou, 2002). In the past decades, researchers have shown the equivalence of paper-based and computer-based tests in terms of testing quality, implying that the development of computer-based testing systems and relevant techniques are worthwhile (Chiou et al., 2009; Hwang, 2003b; Hwang, Chu, et al., 2008). Conventional testing systems represent the learning status of a student by assigning a total score or grade. Such feedback makes the students aware of their learning status through the score or grade; however, this information alone is insufficient for improving their learning performance unless further guidance can be given (Gerber, Grund, & Grote, 2008). This implies that providing learning suggestions for students after testing is an important research issue (Hwang, Tseng, & Hwang, 2008). In recent years, researchers have proposed various approaches for developing adaptive learning systems based on the personal features or learning behaviors of students (Casamayor, Amandi, & Campo, 2009; Cheng, Lin, Chen, & Heh, 2005; Huang, Liu, Chu, & Cheng, 2007; Hwang, Tsai, Tsai, & Tseng, 2008; Manning & Dix, 2008; Offer & Bos, 2009). Furthermore, models or mechanisms for diagnosing student learning problems and providing personalized learning guidance have been presented as well (Bai & Chen, 2008; Huang, Lin, & Cheng, 2009; Hwang, Hsiao, & Tseng, 2003; Lee et al., 2009; Pavlekovic, Zekic-Susac, & Djurdjevic, 2009; Tseng, Sue, Su, Weng, & Tsai, 2007; Tung, Huang, Keh, & Wai, 2009). Among the existing models, the Concept-Effect Relationship (CER) model, which represents the prerequisite relationships among concepts in a course, has been proved to be an effective way of improving the learning performance of students (Hwang, * Corresponding author. Tel.: þ668 40042689; fax: þ662 441 9322. E-mail addresses: [email protected] (P. Panjaburee), [email protected] (G.-J. Hwang), [email protected], [email protected] (W. Triampo), [email protected] (B.-Y. Shih).